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Abstract 

Most learning to rank research has assumed 
that the utihty of different documents is in- 
dependent, which rcsuhs in learned ranking 
functions that return redundant results. The 
few approaches that avoid this have rather 
unsatisfyingly lacked theoretical foundations, 
or do not scale. We present a learning-to- 
rank formulation that optimizes the fraction 
of satisfied users, with a scalable algorithm 
that explicitly takes document similarity and 
ranking context into account. We present 
theoretical justifications for this approach, 
as well as a near-optimal algorithm. Our 
evaluation adds optimizations that improve 
empirical performance, and shows that our 
algorithms learn orders of magnitude more 
quickly than previous approaches. 

1. Introduction 

Identifying the most relevant results to a query is 
a central problem in web search, hence learning 
ranking functions has received a lot of attention 
(e.g., Burges et al., 2005; Chu and Ghahramani, 2005; 
Taylor et al., 2008). One increasingly important goal 
is to learn from user interactions with search engines, 
such as clicks. We address the task of learning a rank- 
ing function that minimizes the likelihood of query 
abandonment (i.e. no click). This objective is par- 
ticularly interesting as query abandonment is a major 
challenge in today's search engines, and is also sensi- 
tive to the diversity and redundancy among documents 
presented. 

We consider the Multi- Armed Bandit (MAB) setting 

Full version of the paper in 27^^ International Conference 
on Machine Learning (ICML), 2010. 



(e.g. Cesa-Bianchi and Lugosi, 2006), which captures 
many online learning problems wherein an algorithm 
chooses sequentially among a fixed set of alterna- 
tives ("arms" or "strategies"). MAB algorithms are 
ideal for online settings with exploration/exploitation 
tradeoffs. While most MAB literature corresponds to 
learning a single best alternative {single-slot MAB), 
MAB algorithms can also be extended to multiple 
slots, e.g. to learn a ranking of documents that 
minimizes query abandonment (Radlinski et al., 2008; 
Streeter and Golovin, 2009). However, most MAB al- 
gorithms arc impractical at web scales. 

Prior work on MAB algorithms has considered exploit- 
ing structure in the strategy space to improve conver- 
gence rates. One particular approach, articulated by 
Kleinberg et al. (2008b) is well suited to our scenario: 
when the strategies (in our case, documents) form a 
metric space and the payoff function satisfies a Lips- 
chitz condition with respect to the metric. The metric 
space allows the algorithm to make inferences about 
similar documents without exploring them. Further, 
they propose a "zooming algorithm" that learns to 
adaptively refine explored regions of the strategy space 
where there is likelihood of higher payoff, and provide 
strong provable guarantees about its performance. 

In web search, a metric space directly models similarity 
between documents.^ Further, one can use additional 
signals. A search user typically scans results top down, 
and clicks on more relevant documents. One can there- 
fore infer the context in which a click happened: the 
skipped documents at higher ranks. To fully exploit 
the context we factor in both conditional clickthrough 
rates and correlated clicks. The former conditions on 
the event that the user skipped a set of documents (as 
suggested by Chen and Karger, 2006), and the latter 
refers to the probability that two documents are both 



^In fact, most offline learning-to-rank approaches also 
rely on similarity between documents, at least implicitly. 
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relevant (or irrelevant) to a given user. 

Our contributions. This paper initiates the study 
of online learning-to-rank in metric spaces. We pro- 
pose a simple learning model that explicitly consid- 
ers correlation of clicks and similarity between doc- 
uments, and admits efficient bandit algorithms that, 
unlike those in prior work on bandit-based learning- 
to-rank, scale to large document collections. We 
study this model both theoretically and einpirically. 
First, we validate the expressiveness of our model 
by providing an explicit construction for a wide fam- 
ily of plausible user distributions which provably fit 
the model. Second, we design several algorithms for 
our model, joining and extending ideas from "ranked 
bandits" (Radlinski et al., 2008), bandits on metric 
spaces (Kleinberg et al., 2008b) and contextual ban- 
dits (Slivkins, 2009).^ Third, we provide provable scal- 
ability guarantees. Finally, we empirically study their 
performance using the above-mentioned construction 
with realistic parameters. 

In a more abstract view, we tackle the problem of us- 
ing side information on document similarity in the on- 
line learning-to-rank setting. We focus on the case 
of "ideally clean" similarity data, with a two-pronged 
high-level question: how to model such data, and how 
to use it algorithmically. We believe that studying the 
"clean" case is useful (and perhaps necessary) to in- 
form and guide the corresponding data-driven work. 

Outline. We define the model in Section 2, validate 
its expressiveness in Section 3, design algorithms in 
Section 4, prove scalability guarantees in Section 5, 
and discuss simulations in Section 6. It is worth noting 
that much of the theoretical (provable) contribution 
concerns setting up various aspects of the model in 
Section 2.2, Section 3, and Section 4.4. To keep the 
flow of the paper, the lengthy proofs from Section 3 
are presented in Section 7. 

2. Multi-slot bandits in metric spaces 

Let us introduce and motivate the online learning-to- 
rank problem that we study in this paper. 

2.1. Problem formalization 

Learning problem. Following (Radlinski et al., 
2008), we are interested in learning an optimally di- 
verse ranking of documents for a given query. We 
model it as a multi-slot MAB problem as follows. Let 



X be a set of documents ("arms"). Each 'user' is 
represented by a binary relevance vector: a function 
TT : X ^- {0,1}. A document x € AT is called "rele- 
vant" to the user if and only if tt{x) = 1. Let Tx be the 
set of all possible relevance vectors. Users come from 
a distribution V on J^x that is fixed but not revealed 
to an algorithm. -^ In each round, the following hap- 
pens: a user arrives, sampled independently from V; 
an algorithm outputs a list of k results; the user scans 
the results top-down, and clicks on the first relevant 
one. The goal is to maximize the expected fraction 
of users who click on a result. Note that in contrast 
with prior work on diversifying existing rankings (e.g. 
Carbonell and Goldstein, 1998), our aim is to directly 
learn a diverse ranking. 

Click probability. The pointwise mean of P is a 
function /i : A — > [0, 1] such that iJ.{x) = E^^-p[7r(a;)]. 
Thus, n{x) is the click probability for document x if it 
appears in the top slot. Each slot i > 1 is examined 
by the user only in the event that all documents in 
the higher slots are not clicked, so the relevant click 
probabilities for this slot are conditional on this event. 
Formally, fix a subset of documents S C X and let 
Zs ^ {Tri') ~ on S} he the event that all documents 
in S are not relevant to the user. Let (T'jZg) be the 
distribution of users obtained by conditioning V on 
this event, and let ^(- \Zs) be its pointwise mean. 

Metric spaces. Throughout the paper, let (A, T)) be 

a metric space.'^ A function i^ : X — )■ R is said to 
be Lip schitz- continuous [L- continuous) with respect 
to (w.r.t.) (A,!?) if 

Hx) - v{y)\ < V{x, y) for ah x, y e A. (1) 

A user distribution V is called L-continuous with re- 
spect to (A, V) if its pointwise mean /i is L-continuous 
w.r.t. (A,X>). 

Document similarity. To allow us to incorpo- 
rate information about similarity between documents, 
we start with the model proposed by Kleinberg et 
al. (2008b), called Lipschitz MAB: an algorithm is 
given a metric space (A, V) w.r.t. which the point- 
wise mean /i is L-continuous.^ This model suffices for 
learning the document at the top position (k = 1). 

However, for lower ranked documents this model is not 



^For more work on (contextual) bandits on metric 
spaces, see (Agrawal, 1995; Kleinberg, 2004; Pandey et al., 
2007; Auer et al., 2007; Kazan and Megiddo, 2007; 
Bubeck et al., 2008; Kleinberg and Slivkins, 2010). 



^Note that this also models users for whom documents 
are probabilistically relevant (Radlinski et al., 2008) 

■'i.e., A is a set and T? is a non-negative symmetric 
function on A x A such that (i) V{x,y) = <=> x = y, 
and (ii) V{x,y) +'D{y,z) > V{x,z) (triangle inequality). 

^One only needs to assume that similarity between any 
two documents x,y is summarized by a number Sx,y such 
that \i-i{x) — ^^{y)\ < S^^y. Then one obtains a metric space 
by taking the shortest paths closure. 



Learning optimally diverse rankings over large document collections 



sufficiently informative since the relevant click prob- 
abilities Ijl{-\Zs) are conditional. We will assume a 
stronger property called conditional L- continuity: 

Definition 2.1. V is called conditionally L- 
continuous w.r.t. (X^V) if the conditional pointwise 
mean iJ,{-\Zs) is L-continuous for all S C X . 

Now, a document x in slot i > 1 is examined only if 
event Zs happens, where S is the set of documents in 
the higher slots, x has a conditional click probability 
Pl{x\Zs)- The function ^(- \Zs) satisfies the Lipschitz 
condition (1), which will allow us to use the machinery 
from MAB problems on metric spaces. 

Formally, we define the k-slot Lipschitz MAB prob- 
lem, an instance of which consists of a triple (X, 'D^V), 
where (X, V) is a metric space that is known to an al- 
gorithm, and P is a latent user distribution which is 
conditionally L-continuous w.r.t. (X, P). 

This formulation includes the "metric-less" fc-slot 
MAB problem from (Radlinski et al., 2008) as a spe- 
cial case in which the metric space is "trivial" : all dis- 
tances are 1. In particular, all algorithms in Section 4 
are well-defined for the "metric-less" version. 

2.2. Correlated relevance 

An alternative notion of document similarity focuses 
on correlated relevance: correlation between the rele- 
vance of two documents to a given user. We express 
"similarity" by bounding the probability of the "dis- 
correlation event" {7r(x) ^ ■K{y)}. Specifically, we con- 
sider conditional L- correlation, defined as follows: 

Definition 2.2. CallV L-corrclated w.r.t. {X,V) if 
Pr [7r(x) ^ iT{y)] < V{x, y) Vx, y ^ X. (2) 

Call V conditionally L-correlated w.r.t. (X,!)) if (2) 
holds conditional on Zs for any S C X , i.e. 

Pr [7rix)^7riy)]<Vix,y) yx,yeX,ScX. 

7r~(7^|Zs) 

It is easy to see that conditional L-correlation implies 
conditional L-continuity. In fact, we show that the two 
notions are essentially equivalent. Namely, we prove 
that conditional L-continuity w.r.t. {X, V) implies con- 
ditional L-correlation w.r.t. {X,2'D). 

Lemma 2.3. Consider an instance (X, I?,P) of the 
k-slot Lipschitz MAB problem. Then the user distri- 
bution V is conditionally L-correlated w.r.t. (X,2'D). 

Proof. Fix documents x,y £ X and a subset S d X. 
For brevity, write "a: = 1" to mean "7r(a;) = 1" , etc. 



We claim that 

PT[x = l/\y = 0\Zs]<V{x,y). (3) 

Indeed, consider the event Z = Zg^^yy Applying the 
Bayes theorem to {P\Zs), we obtain that 

^^ix\Z)^Pr[x^l\{y^O}AZs] 
Pr[x^ lAy^O\Zs] 



P4y = \Zs] 



(4) 



On the other hand, since n{y\Z) = 0, by conditional 
L-continuity it holds that 

fi{x\Z) = |^(x|Z) - ijiy\Z)\ < V{x,y), (5) 

so the claim (3) follows from (4) and (5). 
Likewise, Pr[.T = A y = 1 \Zs\ < D{x, y). Since 

{tt{x) ^ 7r(y)} = {x = 1 A jy = 0} U {x = A y = 1}, 

it follows that Pr[7r(x) ^ 7r(y) \Zs\ < 2D{x,y). D 

2.3. Metric space: a running example 

Web documents are often classified into hierarchies, 
where closer pairs are more similar.^ For evaluation, 
we assume the documents X fall in such a tree, with 
each document x G X a leaf in the tree. On this tree, 
we consider a very natural metric: the e-exponential 
tree metric: the distance between any two tree nodes 
u, V is exponential in the height of their least common 
ancestor, with base e G (0, 1): 



V{u,v) 



ex e 



height(LCA(M,D)) 



(6) 



for some constant c. However, our algorithms and 
analyses extend to arbitrary metric spaces. 

3. Expressiveness of the model 

Our approach relies on the conditional L-continuity 
(equivalently, conditional L-correlation) of the user 
distribution. How "expressive" is this assumption, i.e. 
how rich and "interesting" is the collection of problem 
instances that satisfy it? While the unconditional L- 
continuity assumption is usually considered reasonable 
from the expressiveness point of view, even the uncon- 
ditional L-correlation (let alone the conditional one) is 
a very non-trivial property about correlated relevance, 
and thus potentially problematic. A related concern 
is how to generate a suitable collection of problem in- 
stances for simulation experiments. 

In this section we address both concerns by defining 
a natural (albeit highly stylized) generative model for 



E.g., the Open Directory Project http://dnioz.org/ 
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the user distribution, which we then use in the ex- 
periments in Section 6. Given a tree metric space 
{X, V) and the desired pointwise mean /x, the genera- 
tive model provides a rich family of user distributions 
that arc conditionally L-continuous w.r.t. (X, c2?), for 
some small c. We develop the generative model in 
Section 3.1. We extend this result in two directions: 
to arbitrary metric spaces (Section 3.2) and to distri- 
butions over conditionally L-continuous user distribu- 
tions (Section 3.3). 

3.1. Bayesian tree network 

The generative model is a tree-shaped Bayesian net- 
work with 0-1 "relevance values" 7r(-) on nodes, where 
leaves correspond to documents. The tree is essentially 
a topical taxonomy on documents: subtopics corre- 
spond to subtrees. The relevance value on each sub- 
topic is obtained from that on the parent topic via a 
low-probability mutation. 

The mutation probabilities need to be chosen so as 
to guarantee conditional L-continuity and the desired 
pointwise mean /i. It is fairly easy to derive a necessary 
and sufficient condition for the pointwise mean, and a 
necessary condition for conditional L-continuity. The 
latter condition states that the mutation probabilities 
need to be bounded in terms of the distance between 
the child and the parent. The hard part is to prove 
that this condition is sufficient. 

Let us describe our Bayesian tree network in detail. 
The network inputs a tree metric space (X, T)) and 
the desired pointwise mean /x, and outputs a rele- 
vance vector TT. Specifically, we assume that docu- 
ments are leaves of a finite rooted edge- weighted tree 
Td with node set V and leaf set X <zV. Let V be the 
(weighted) shortest-paths metric on V . The desired 
pointwise mean /i can be any function ^ : X ^^ [a, i], 
a > 0, that is L-continuous w.r.t. (X, 2?). We show 
(see Section 7.1) that /i can be extended from X to y 
preserving the range and L-continuity. 

The Bayesian network itself is very intuitive. We pick 
7r(root) G {0, 1} at random with a suitable expecta- 
tion, and then proceed top-down so that the child's 
click is obtained from the parent's click via a low- 
probability mutation. The mutation is parameterized 
by functions goj^i : ^ — >■ [0, 1], as described in Algo- 
rithm 1. These parameters let us vary the degree of 
independence between each child and its parent, re- 
sulting in a rich family of user distributions. 

To complete the construction, it remains to define the 
mutation probabilities qo,qi- Let V be the resulting 
user distribution. It is easy to see that /i is the point- 



Algorithm 1 User distribution for tree metrics 
Input: Tree (root r, node set V); /x(r) S [0, 1] 

mutation probabilities qo,qi :V^[0, 1] 
Output: relevance vector tt : y — > {0, 1} 

function AssignClicks(tree node v) 
b <— 7r(u) 
for each child u of u do 

1 — fe w/prob qb{u) 
b otherwise 

AssignClicks(u) 

Pick 7r(r) e {0, 1} at random with expectation fi{r) 
AssignClicks(r) 



7r(it) 



wise mean oiV on V if and only if 

m(") = (1 - M(t')) qo{u) + m(i')(1 - 91 (w)) (7) 

whenever u is a child of v. (For sufficiency, use induc- 
tion on the tree.) Further, letting qb ~ qb{u) for each 
bit b € {0, 1}, note that 

Pr[7r(u) ^ 7r(u)] = ^(u) gi + (1 - ^i{v)) qo 

= fJ'{v){qo + qi + {l- 2/i(i;)) qo 
> fi{v){qo + qi). 

Thus, if V is L-correlated w.r.t. {X, T>) then 

qa{u) + qi(u)<V{u,v)/^i{v). (8) 

We show that (7-8) suffices to guarantee conditional 
L-continuity. For a concrete example, one could define 

(qo(^), gi(w)) = <( ) , ^^;^, \ ^^ '-^^ ' (9) 

^ / Mu)-M(.) ^ Q^ otherwise. 

The provable properties of Algorithm 1 are summa- 
rized in the theorem below. It is technically more con- 
venient to state this theorem in terms of L-correlation 
rather than L-continuity. 

Theorem 3.1. Let V be the shortest-paths metric of 
an edge-weighted rooted tree with a finite leaf set X. 
Let fi : X ^ [a,^], a > be L-continuous w.r.t. 
{X,V). Suppose qo,qi:V -^ [0, 1] satisfy (7-8). 

Let V be the user distribution constructed by Algo- 
rithm L Then V has pointwise mean fi and is con- 
ditionally L-correlated w.r.t. (X, 3 2?^) where 



T^f^{x,y) -T^{x,y) 



a ' ti{x}-\-p,{y) 



(10) 



Remark. The theorem can be strengthened by replac- 
ing Vfj, with the shortest- paths metric induced by 2?^. 
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The proof of Theorem 3.1 is a key theoretical contri- 
bution of this work, and by far the most technical one. 
Below we provide a proof sketch. To keep the flow of 
the paper, the full proof is presented in Section 7.2. 

Proof Sketch. As we noted above, the statement about 
the pointwise mean trivially follows from (7) using in- 
duction on the tree. In what follows we focus on con- 
ditional L-corrclation. 

Fix leaves x,y & X and a subset S C X. Let z be 
the least common ancestor of x and y. Recall that in 
Algorithm 1 the value of 7r(-) at each node is a random 
permutation of that of its parent. We focus on the 
event £ that no mutation happened on the 2 — > a; and 
z ^ y paths. Note that £ implies ■k{x) ~ 7r(y) = T^iz). 
Therefore 



FT[Trix)^7r{y)\Zs]<Pr[£\Zs] 



(11) 



where £ is the negation of £. Intuitively, £ is a low- 
probability "failure event". The rest of the proof is 
concerned with showing that Pr[5 \Zs] < 3'Df^{x,y). 

First we handle the unconditional case. We claim that 



PY[£]<V^{x,y). 



(12) 



Note that (12) immediately implies that V is L- 
corrclated w.r.t. (A", 2?^). This claim is not very dif- 
ficult to prove, essentially since the condition (8) is 
specifically engineered to satisfy the unconditional L- 
correlation property. We provide the proof in detail. 

Let w G argmin„gp m(")j where Pxy is the a; — > y 
path. Let {z = a;o,a;i, ... , a;„ = x) be the z — > a; 
path. For each « > 1 by (8) the probability of having 
a mutation at Xi is at most ^{xiyXi-i)/ fi{w), so the 
probability of having a mutation on the z — ?> a; path 
is at most 2?(a:, z)/ ijl{w). Likewise for the z -^ y path. 
SoPT[£]<Vix,y)/fiiw)<^. 

It remains to prove that 

Pr[^1<^(^.y)MJ)W- ^13) 

Indeed, by L-continuity it holds that 

^{w) > /i(a;) — 'D{x, w), 
fJ-{w) > fJ,{y)-V{y,w). 

Since I?(a;, y) = I'(x, w) + Vi^y, w), it follows that 

fi{w) > t-(^)+^^iv)-^(--v) , (14) 

Now, either the right-hand side of (14) is at least 
Kx)+tJ.(v) ^ Qj, -(-j^g right-hand side of (13) is at least 1. 
In both cases (13) holds. This completes the proof of 
the claim (12). 



The conditional case is much more difficult. We handle 
it by showing that 



Pr[£|Zs]<3Pr[f]. 



(15) 



In fact, (15) holds even if (8) is replaced with a much 
weaker bound: ina.x{qo{u), qi{u)) < ^ for each u. 

The mathematically subtle proof of (15) can be found 
in Section 7.2. The crux in this proof is that event Zs 
is more likely if document z is not relevant to the user: 

Pr[Zs|z==0] >Pr[Z5|2 = l]. D 

3.2. Arbitrary metric spaces 

We can extend Theorem 3.1 to arbitrary metric 
spaces using prior work on metric embeddings. Fix 
an A^-point metric space {X, V) and a function fi : 
X — >■ [a, i] that is L-continuous on (X,!)). It 
is known (Bartal, 1996; Fakcharoenphol et al., 2004) 
that there exists a distribution T^trcc over tree metric 
spaces {X,T) such that 'D{x,y) < T{x,y) and 

Er~n..ee [r{x, y)] < cV{x, y) \/x, y & X, 

where c = 0(log A^).^ 

The construction is simple: first sample a tree metric 
space {X,T) from T^trcc; then independently generate 
a user distribution Vj- for {X,T) as per Algorithm 1. 
Let V be the resulting aggregate user distribution. 

Theorem 3.2. The aggregate user distribution V has 
pointwise mean fj. and is conditionally L-correlated 
w.r.t. (AT, 3cl?^), where V^ is given by 



D^ix,y) = D{x,y) min (^ 



a ' fi{x)+ii{y) 



Proof. The function fi is L-continuous w.r.t. each tree 
metric space {X,T), so by Theorem 3.1 user distri- 
bution Vj- has pointwise mean /x and is conditionally 
L-correlated w.r.t. (A', 3 7^). It follows that the ag- 
gregate user distribution P has pointwise mean /i, and 
moreover for any x, y G A" and S G X we have 

Pr [7r{x)^7r{y)\Zs] 



< Er~n,co 

<Er^P_ [3%{x,y)] 

< ic'D^{x,y). 



Pr [t:{x) ^ T,{y) \Zs] 



D 



^For point sets in a d-dimensional Euclidean space one 
could take c — 0{d\og -), where e is tlie minimal distance. 
In fact, this result extends to a much more general family 
of metric spaces - those of doubling dimension d. Dou- 
bling dimension, the smallest d such that any ball can be 
covered by 2^* balls of half the radius, is a well-studied 
concept in the theoretical computer science literature, e.g. 
sec (Gupta et al., 2003). 
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3.3. Distributions over user distributions 

Let us verify that a distribution over conditionally 
L-continuous user distributions is conditionally L- 
continuous. Apart from being a natural and desirable 
property, this result considerably extends the family 
of user distributions for which we have conditional L- 
continuity guarantees. 

Lemma 3.3. Let V be a distribution over countably 
many user distributions Vi that are conditionally L- 
continuous w.r.t. a metric space (X, P). Then V is 
conditionally L-continuous w.r.t. [X^D). 

Proof. Let /i and fii be the (conditional) pointwise 
means oiV and Vi, respectively. Formally, let us treat 
each Vi as a measure, so that Vi{E) is the probability 
of event E under Vi. Let V = J2i 1i ^i' where {qi} are 
positive coefRcients that sum up to 1. Fix documents 
x,y £ X and a subset S C X. Then 



fi{x\S)=V{x = l\Zs)^ 



Vix = lAZs) 



V{Zs) 
Y.^q,V,{x^lAZs) 
V{Zs) 

J2,qi v^{Zs) ^i^{x\Zs) 

V{Zs) 



It follows that 



\^,{x\s) - ^l{y\s)\ 

^ Ez9» 'PjjZs) {^l^{x\Zs) - ^l^{y\Zs)) 

V{Zs) 
j:^q,V,iZs)D{x,y) 



< 



V{Zs) 



<D{x,y). 
4. Algorithms 



D 



In this section we present algorithms for the fc-slot 
Lipschitz MAB problem defined in Section 2. In Sec- 
tions 4.1-4.2 wc describe several algorithms from prior 
work on, respectively, multi-slot MAB and Lipschitz 
MAB, and their adaptations to our setting. Interest- 
ingly, one can combine algorithms from prior work in a 
modular way, which we make particularly transparent 
by putting forward a suitable naming scheme. Then 
in Section 4.3 we review the contextual bandit setting 
and one particular algorithm for this setting. Finally, 
in Section 4.4 we represent our setting as a (multi-slot) 
contextual MAB problem, and use this representation 
to design two new algorithms that select documents 
slot- by-slot (top-down) , and explicitly treat the higher 
selected documents as a context for the current slot. 

As mentioned earlier, the "metric- aware" algorithms 
presented here are well-defined for arbitrary metric 



spaces, but for simplicity we present them for a spe- 
cial case in which documents are leaves in a document 
tree rj with an e-exponential tree metric. In all these 
algorithms, a subtree is chosen in each round. Then a 
document in this subtree is sampled at random, choos- 
ing uniformly at each branch. 

4.1. Prior work: ranked bandits 

Letting Bandit be some algorithm for the MAB prob- 
lem, the "ranked" bandit algorithm RankBandit for 
the multi-slot MAB problem is defined as follows 
(Radlinski et al., 2008). We have k slots (i.e., ranks) 
for which we wish to find the best documents to 
present. In each slot i, a separate instance Ai of 
Bandit is created. In each round these instances select 
the documents to show independently of one another. 
If a user clicks on slot i, then this slot receives a payoff 
of 1, and all higher (i.e., skipped) slots j < i receive a 
payoff of 0. For slots j > i, the state is rolled back as 
if this round had never happened (as if the user never 
considered these documents). If no slot is clicked, then 
all slots receive a payoff of 0. 

In (Radlinski et al.. 2008), this approach gives rise to 
algorithms RankUCBl and RankEXPS, based on MAB 
algorithms UCBl and EXP3 (Auer et al., 2002a, 2002b). 
EXP3 is designed for the adversarial setting with no 
assumptions on how the clicks are generated, which 
translates into concrete provable guarantees for Rank- 
EXPS. UCBl is geared towards the stochastic setting 
with i.i.d. payoffs on each arm, although the per- 
slot i.i.d. assumption breaks for slots i > 1 because 
of the influence of the higher slots. Nevertheless, in 
small-scale experiments RankUCBl performs much bet- 
ter than RankEXPS (Radlinski et al., 2008). 

In UCBl-style algorithms, including the zooming al- 
gorithm, one can damp exploration by replacing the 
41og(r) factor in (16) with 1. Such change effectively 
makes the algorithm more optimistic; it was found 
beneficial for RankUCBl by Radlinski et al. (2008). We 
will denote this version by appending '-[-' to the algo- 
rithm's name, e.g. RajikUCBl+. We will see that it can 
also greatly improve average performance here. 

4.2. Prior work: using the metric space 

Both above algorithms are impractical when there are 
too many documents to explore them all. To avoid this 
challenge, we can exploit the similarity information 
provided by the metric space in our setting. Since 
the payoff function /i(-) is an L-continuous function on 
(AT, 2?), we can use the (single-slot) bandit algorithms 
that are designed for the Lipschitz MAB problem to 
improve the speed of convergence of the RankedBandit 
approach. 
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Algorithm 2 "Zooming algorithm" in trees 
initialize (document tree t^): 
yl<— 0; activate(root(Td)) 

activate( u G nodes(Td) ): 

Main loop: 

u ^ argmax„g_^ ^ + 2 rad(w) 
"Play" a random document from subtree(it) 
r{u) <— r{u) + {reward}; n{u) <— n{u) + 1 
if rad(w) < W(w) then 

deactivate u: remove u from A 

activate all children of u 

The meta- algorithm GridBandit (Kleinberg, 2004) is 
one such algorithm. It proceeds in phases: In phase i, 
the depth-i subtrees are treated as "arms" , and a fresh 
copy of Bcindit is run on these arms.® Phase i lasts for 
fce~^' rounds, where k is the number of depth-i sub- 
trees. This meta-algorithm (coupled with an adver- 
sarial MAB algorithm such as EXP3) is the only prior 
algorithm that takes advantage of the metric space 
in the adversarial setting. Following (Radlinski et al., 
2008), we expect GridEXP3 to be overly pessimistic for 
our problem, trumped by the corresponding stochastic 
MAB approaches such as GridUCBl. 

The "zooming algorithm" (Kleinberg et al., 2008b, Al- 
gorithm 2) is a more efficient version of GridUCBl: in- 
stead of iteratively reducing the grid size in the entire 
metric space, it selectively refines the grid in promising 
areas. It maintains a set A of active subtrees which col- 
lectively partition the leaf set. In each round the active 
subtree with the maximal index is chosen. The index 
of a subtree is (assuming stochastic payoffs) the best 
available upper confidence bound on the click proba- 
bilities in this subtree. It is defined via the confidence 
radius^ given (letting T be the time horizon) by 



rad(-) ^ V41og(T)/(l + #samples(-)). 



(16) 



The algorithm "zooms in" on a given active sub- 
tree u (de-activates u and activates all its children) 
when rad(u) becomes smaller than its width W(u) = 
e'^'^P'^^^") = maxa;^2,/g„I'(x,x'). The "ranked zooming 
algorithm" will be denoted RankZoom. 

4.3. Prior work: contextual bandits 

Our subsequent algorithms leverage prior work on con- 
textual MAB. The relevant contextual MAB setting is 



® As an empirical optimization, previous events can also 
be replayed to better initialize later phases. 

®The meaning of rad(-) is that the sample average is 
within ±rad(-) from the true mean with high probability. 



Algorithm 3 ContextZoom in trees 

initialize (document tree rj, context tree Tc): 
yl<— 0; activate( root(rd), root(Tc) ) 

activate ( u G nodes(r(j), u^. G nodes(rc) ): 
v4-i— ^U {(m, Uc)}; n(M,Uc)-i— 0; r(u,Mc)-i— 

Main loop: 

Input a context h G nodes(Tc) 

(m,Uc)^ argmax W(uxuc)+4^^-|-rad(w,Uc) 

"Play" a random document from subtree(u) 
r{u,Uc) ^— r (M,Uc)-|-{reward}; n(u,Mc) ^— n(M,Uc)+l 
if rad(u,Uc) < W(u,Uc) then 
deactivate (u,Mc): remove (u,Wc) from A 
activate all pairs (child(M), child(uc)) 



as follows: in each round nature reveals a context h, 
an algorithm chooses a document x, and the resulting 
payoff is an independent {0, 1} sample with expecta- 
tion ^{x\h). Further, one is given similarity informa- 
tion: metrics V and 2?c on documents and contexts, 
resp., such that for any two documents x,x' and any 
two contexts h, h' we have 



|/i(2:|/i) ~ ^i{x'\h')\ <V{x,x')+Vc{h,h') 



(17) 



Let Ac be the set of contexts, and Adc = A x Ac be 
the set of all (document, context) pairs. Abstractly, 
one considers the metric space (Adc^^dc), henceforth 
the DC-space, where the metric is 

Pdc((a;, h), ix', h')) = P(a;, x') + V,{h, h'). 

We will use the "contextual zooming algorithm" 
(ContextZoom) from (Slivkins, 2009). This algorithm 
is well-defined for arbitrary Pdc, but for simplicity 
we will state it for the case when T) and 2?c are e- 
exponcntial tree metrics. 

Let us assume that documents and contexts are leaves 
in a document tree t^ and context tree Tc, respectively. 
The algorithm (see Algorithm 3 for pseudocode) main- 
tains a set A of active strategies of the form {u,Uc), 
where u is a subtree in Td and Uc is a subtree in Tc . At 
any given time the active strategies partition Adc. In 
each round, a context h arrives, and one of the active 
strategies (u, Uc) with /i G Uc is chosen: namely the one 
with the maximal index, and then a document x € u 
is picked uniformly at random. The index of (u, Uc) is, 
essentially, the best available upper confidence bound 
on expected payoffs from choosing a document x G w 
given a context h £ Uc'. as per (Slivkins, 2009), with 
high probability it holds that 



index(u, Uc) > ^{x\h), Wx G u, h E Uc 



(18) 
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The index is defined via sample average, confidence 
radius (16), and "width" W(u xuc)- The latter is an 
upper bound on the diameter of the "rectangle" uxu^ 
in the DC-space: 



W(u,Uc)> max V{x,x')+Vc{h,h'). (19) 

x,x' Gu, h^h' f^Uc 



The (de) activation rule ensures that the active strate- 
gies form a finer partition in the regions of the DC- 
space that correspond to higher payoffs and inore fre- 
quently occurring contexts. 

4.4. New approach: ranked contextual bandits 

We now present a new approach in which the upper 
slot selections are taken into account as a context. 

The slot algorithms in the RankBandit setting can 
make their selections sequentially. Then without loss 
of generality each slot algorithm Ai knows the set S 
of documents in the upper slots. We propose to treat 
5 as a "context" to Ai- Specifically, Ai will assume 
that none of the documents in S is clicked, i.e. event 
Zs happens (else the i-th slot is ignored by the user). 
For each such round, the click probabilities for Ai are 
given by /i(- \Zs), which is an L-continuous function 
on iX,V). 

Let us specify a suitable metric Vc on contexts S G X: 



V,iS,S')^Amfj:[l,Vix,,x',), 



(20) 



where the infimum is taken over all n e N and over all 
n-element sequences {xj} and {x'A that enumerate, 
possibly with repetitions, all documents in S and S' . 

We show that Vc satisfies (17) via the following lemma. 

Lemma 4.1. Let {X,Ty,V) be an instance of the k-slot 
Lipschitz MAB problem. Fix x & X and S,S' C X. 
Define Vc{S, S") by (20). Then 

\^i{x\Zs)-^Ji{x\Zs>)\<v,{s,s'). 



Proof. For shorthand, let us write 

a[x\S) = l-^(x|Zs), 
a{x\S,y)^a{x\SiJ{y}). 

First, we claim that for any y (z X and y' £ S 

\a{x\S,y)-a{x\S,y')\<4V{y,y'). (21) 



can re- write the left-hand side of (21) as 

(j{y\S,x) a{y'\S,x) 



LHS(21) = a{x,S) 



aiy\S) a{y'\S) 



<a(.,5)P(,,,0 "^"'lt"/fof^ (22) 



-Diy^y') 



aiy\S)a{y'\S) 
a{x\S) +a{x\S,y) 



a{y'\S) 



<2I?(y,2/')- 



In (22), we have used the L-continuity of cr(-|S') and 
a{-\S,x). To achieve the constant of 2, it was crucial 
that y' G 5, so that c7{y'\S) = 1. This completes the 
proof of (21). 

Fix some n G N and some n-element sequences {xi} 
and {x'^} that enumerate, possibly with repetitions, all 
values in S and S", respectively. Consider sets Si ~ 
{x[ , ... , a:-} U {.Ti+i , . . . , Xn}, 1 < i < n- I, and 
let So ~ S and Sn+i = S". To prove the lemma, it 
suffices to show that 

\aix\S,) - aix\S,+i)\ < 4I?(a;,+i, x'^+^) (23) 

for each i < n. To prove (23), fix i and let y = Xi+i 
and y' = x-^^^. Note that Si U {y'} = Si+i U {y}, call 
this set S* . Then using (21) (note, y G Si and y' £ S^) 
we obtain 

\<j{x\S,)~a{x\S*)\^Hx\S,,y)-<j{x\S,,y')\ 
<2V{y,y'), 
|cr(a;|S'i+i) -cr(x|5'*)| = \cr{x\Si+i,y') - a{x\Si+i,y)\ 
<2V{y,y'), 



which implies (23). 



D 



Indeed; noting that ij{x\S^y) — a{y\S^x) 



(j{x\S) 



Having defined Vc, we can use any contextual MAB 
algorithm for Ai. We will use ContextZoom. 

Let us supply the missing details as they apply to the 
{i + l)-th slot, i > 1. The contexts are unordered i- 
tuples of documents. Given a document tree tj, let 
us define context tree Tc as follows. Depth-^ nodes of 
Tc are unordered z-tuples of depth-Z nodes from tj, 
and leaves arc contexts. The root of Tc is (r...r), 
where r = root(rd). For each internal node u^ ~ 
(ui . . . Ui) of Tc, its children are all unordered tuples 
{vi . . -Vi) such that each Vj is a child of Uj in tj. This 
completes the definition of Tc. Letting u and u^ be 
level-/ subtrees of Td and Tc, resp., it follows from (20) 
that Vc[S, S') < 4i e' for any contexts S, S' S u^. Thus 
setting W(w xuc) = e'(4i + 1) satisfies (19). 

We will use ContextZoom (with Tc and W(m, Uc) as 
above) for slots i > 2; for slot 1, contexts are empty, 
so ContextZoom reduces to Algorithm 2. The resulting 
"ranked" algorithm is called RankContextZoom. 
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Exploiting correlations. We can further exploit the 
conditional L-continuity. In the analysis of Context- 
Zoom, the index can be decreased, improving perfor- 
mance, as long as (18) holds. Fix context S C X. 
Since ^i{y\Zs) = for any y G S, 

^lix\Zs) ^ Hx\Zs) - fiiy\Zs)\ <V{x,y), Vy e 5 
hence fi{x\Zs) < 'D{x,S) = minygg 'D{x,y). (24) 

In other words, if document x is close to some docu- 
ment in S, the event Zs limits the conditional proba- 
bility fi{x\Zs)- Using (24) we might be able to decrease 
the index of a strategy without violating (18). Thus, 
we "upgrade" RankContextZoom with the following 
correlation rule: for each active strategy {u, Uc), where 
M is a subtree of documents, and u^. is a set of contexts, 
cap the index of each (m, Uc) at T^^u, S). 

We also define RankCorrZoom, a version of RankZoom 
which uses a similar "correlation rule": for each slot, 
cap the index of each active subtree u at 'D{u, S). The 
intuition is (again) that decreasing the index prov- 
ably increases performance of the zooming algorithm 
as long as the index is a valid upper bound on expected 
payoffs. We view RankCorrZoom as a light-weight way 
version of RankContextZoom: a low-overhead (but per- 
haps limited) way to use contexts inside RankZoom. 

5. Provable scalability guarantees 

Here we summarize the relevant provable guarantees 
from prior work, and apply them to our multi-slot algo- 
rithms. The purpose is two-fold: to provide intuition 
behind these algorithms, and to prove their scalability. 
We also provide an improved convergence result. 

5.1. Prior work 

Provable guarantees for single-slot MAB algorithms 
are usually expressed via regret w.r.t. a benchmark: 
the best arm in hindsight. Regret R{T) of an algo- 
rithm is the expected payoff of the benchmark in T 
rounds minus that of the algorithm. We will discuss re- 
gret bounds for (i) the "plain" (unstructured) n-armed 
bandits, (ii) bandits on metric spaces, and (iii) con- 
textual bandits. We will consider "adversarial pay- 
offs" (with "oblivious", non-adaptive adversary) and 
"stochastic payoffs" (with i.i.d. payoffs for each arm). 

n-armed bandits. EXP3 (Auer et al., 2002b) achieves 
regret R{T) ~ 0{y/nT) against an oblivious (non- 
adaptive) adversary. In the stochastic setting, 
UCBl (Auer et al., 2002a) performs much better, with 
logarithmic regret for every fixed ^. Namely, each 
arm x G X contributes only 0(logr)/A(x) to regret, 
where A(x) = max/^(-) — /i(x). Noting that the total 
regret from playing arms with A(-) < 6 can be a priori 



upper-bounded by ST, we bound regret of UCBl as: 



RiT) 



min I 6T 
<5>o 



V- 0(logT)\ /2^^ 

2^xeX:A{x)>5 A{x) J ■ y^'') 



Note that (25) depends on /i. In particular, if A(-) > S 
then R{T) = 0{j logT). However, for any given 
T there exists a "worst-case" pointwise mean fiT 
such that R{T) = Q[y/nT) in (25), matching EXP3. 
The above regret guarantees for EXP3 and UCBl are 
optimal up to constant factors (Auer et al., 2002b; 
Kleinberg et al., 2008a). 

Bandits on metric spaces. For the Lipschitz MAB 
problem (Kleinberg, 2004; Kleinberg et al., 2008b), re- 
gret guarantees are independent of the number of 
arms. Instead, they depend on the covering properties 
of the metric space [X,!)). The crucial notion here is 
the covering number Nr{X), defined as the minimal 
number of balls of radius r sufficient to cover X . For 
metrics with "uniform density" (informally: without 
"denser" and "sparser" regions) the covering numbers 
can be usefully summarized by a single number called 
the "covering dimension". Given a constant a, the 
covering dimension CovDim(X, 2?) is defined as 



mi{d>0:Nr<ar-'' Vr > 0}, 



(26) 



with Nr = Nr{X)}'^ In particular, for an arbitrary 
point set in Mf^ under the standard {£2) distance, the 
covering dimension is d, for some a = 0{1). For an e- 
exponential tree metric with maximal branching factor 
6, the covering dimension is d = \ogi/i:{b), with a — 1. 

Against an oblivious adversary, GridEXPS has regret 

i?(r) = d(ar(''+i)/(''+2)), (27) 

where d is the covering dimension of (X, P). 

For the stochastic setting, GridUCBl and the zoom- 
ing algorithm have better ^-specific regret guarantees 
similar to (25) for UCBl. In fact, we will state the 
guarantees for all three algorithms in a common form. 
Consider payoff scales 5 = {2* : i € N}, and for each 
scale r e S define Xr = {x e X : r < A{x) < 2r}. 
Then regret (25) of UCBl can be restated as 



RiT) = fi^^[ST + EreS:r>s 



N, 



{S,r} 



O(logT) 



(28) 



where N(^s,r) = \Xr\- In what follows, algorithms 
will have //-specific regret bounds of the form (28), 
which will imply simpler (but sometimes less efficient) 
bounds of the form (27) ( "dimension- type" bounds). 

Now, it follows from the analysis in (Kleinberg, 2004; 
Kleinberg et al., 2008b) that regret of GridUCBl is (28) 



We will keep the constant a implicit in the notation. 
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with Nf^s.r) = Ns{Xr). For the worst-case /x one could 
have Ns{Xr) ~ Ns{X), m which case the /x-specific 
bound essentially reduces to (27). 

For the zooming algorithm, the ^-specific bound can 
be improved to (28) with N(^gj.-j = Nr{Xr)- This im- 
plies an improved dimension-type bound: (27) with 
a different, smaller d called the zooming dimension, 
which is defined as (26) with Nr = Nr{Xr). Note that 
the zooming dimension depends on the triple (X, I?, /x) 
rather than on the metric space alone. 

The zooming dimension can be as high as the cover- 
ing dimension for the worst-case /x, but can be much 
smaller (e.g., d = 0) for "benign" problem instances, 
see (Kleinbcrg et al., 2008b) for further discussion. 
For a simple example, suppose an e-exponential tree 
metric has a "high-payofF' branch and a "low-payofF' 
branch with respective branching factors 6 <C 6'. Then 
the zooming dimension is log]^/j(6), whereas the cov- 
ering dimension is logj^ /^(fo'). 

Contextual bandits. For the contextual MAB prob- 
lem, the guarantees are in terms of contextual regret, 
which is regret is w.r.t. a much stronger benchmark: 
the best arm in hindsight for every given context. 

Regret guarantees for ContextZoom focus on the DC- 
space (Xdc,2?dc) (see Section 4.3 for notation). We 
partition X^^ according to payoff scales r G 5: 

A(x\h) = max/i(-|ft,) — iJ,{x\h), x E X,h E Xc- 
Xdc,r = {ix,h) eXdc-. r< A{x\h) < 2r}. 

Then contextual regret of ContextZoom can be 
bounded by (28) with A^(a-,,.) = Nr{Xdc,r), where 
Nr{-) now refers to the covering numbers in the DC- 
space. To derive the corresponding dimension-type 
bound, define the contextual zooming dimension ddc 
of the problem instance (Xdc,2?dc, A*) as (26) with 
Nr = Nr{Xdc,r)- Thcn one obtains (27) with d = ddc- 
It is easy to see that 



CovDim(Xc,I?c) < i^dc < CovDim(Xdc,2?dc 



(29) 



The upper bound in (29) corresponds to the worst-case 

/i such that Nr{Xdc.r) = Nr{Xdc)- 

The /i-specific bound for ContextZoom can be im- 
proved by taking into account "benign" context ar- 
rivals: effectively, one can prune the regions of Xc that 
correspond to infrequent context arrivals, see (Slivkins, 
2009) for details. This improvement can be especially 
significant if CovDim(Xc,I'c) > CovDim{X,'D). 
Discussion 5.1. ContextZoom is parameterized by the 
time horizon, denote it Tq. To obtain a regret bound 
that holds for each round T < Tq, replace the log(T) 
term in (28) with log(ro) (which then implies the cor- 
responding dimension- type bound). In fact, this regret 



bound holds with probability at least 1 — Tq^, so that 
one can take the Union Bound over all rounds T < Tq. 
Moreover, the anytime version of ContextZoom can be 
defined via the doubling trick: in each phase i S N, run 
a fresh instance of the algorithm for 2* rounds. This 
version is well-defined (and, essentially, has the same 
regret bounds) for any time T. This discussion also 
applies to the zooming algorithm. 

Multi-slot bandits. Letting T be the number of 
rounds and OPT be the probability of clicking on the 
optimal ranking, algorithm RankBandit achieves 

E[#clicks] > (1 - i) T X OPT - fc R{T), (30) 

where RiT) is any upper bound on the 
context-oblivious regret for Bandit in each 
slot (Radlinski et al., 2008). 

In the multi-slot setting, performance of an algorithm 
up to time t is defined as the time-averaged expected 
total number of clicks. We will consider performance 
as a function of t. Assuming R{T) = o{T) in (30), 
performance of RankBandit converges to or exceeds 
(1 - ■i)DPT. Convergence to (1 - ■i)OPT is proved to 
be worst-case optimal. Thus, as long as R{T) scales 
well with T^documents (e.g., as in (27)), Radlinski et 
al. (2008) interpret (30) as a proof of an algorithm's 
scalability in the multi-slot MAB setting. 

RankBandit is presented in (Radlinski et al., 2008) as 
the online version of the greedy algorithm: an offline 
fully informed algorithm that selects documents greed- 
ily slot by slot from top to bottom. The performance of 
this algorithm is called the greedy optimum,^^ which is 
equal to (1 — ^) OPT in the worst case, but for "benign" 
problem instances it can be as good as OPT. The greedy 
optimum is a more natural benchmark for RankBandit 
than (l — i) OPT. Surprisingly, results w.r.t. this bench- 
mark arc absent in the literature. 

5.2. Our results 

Regret. We show that RankGridEXPS and Rank- 
ContextZoom scale to large document collections, in 
the sense that they achieve (30) with R{T) that does 
not degenerate with ^documents. 

Theorem 5.2. Let [X^V^V) he an instance of the 
k-slot Lipschitz MAB problem. Then 

(a) RankGridEXPS achieves regret (30), where R(T) 
is given by (27) with d = CovD±m{X,T>). 

(b) Let ddc be the contextual zooming dimension for 
slot i. Then RankContextZoom achieves (30), 
where R(T) is given by (27) with d = ddc- 



If due to ties there are multiple "greedy rankings", 
define the greedy optimum via the worst of them. 
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Proof. The respective regret bounds for RankGridEXPS 
and RankContext Zoom plug into (30). D 

Remark 5.3. We do not have any provable guarantees 
for RcinkGridUCBl, RankZoom and RankCorrZoom be- 
cause non-contextual regret bounds for the single-slot 
stochastic setting do not plug into (30). 
Discussion 5.4. Contrary to our intuition, our prov- 
able guarantee for RankContextZoomis worse than the 
one for RankGridEXPS. Indeed, by (29) it holds that 

ddc > CovDiin(Xc,I?c) = fc x CovDim(X,2?), 

where {Xc,T>c) is the context space for slot k. 

However, we believe that the bound in part (b) is very 
pessimistic. It builds on the result for ContextZoom 
that contextual regret for slot i is bounded by (27) 
with d = ddc- This result does not take into account 
that for a given slot, contexts S d X may gradually 
converge over time to the greedy optimum and thus 
become "low-dimensional".^^ We believe this effect is 
very important to the performance RankContextZoom. 
In particular, it causes RankContextZoom to perform 
much better than RankGridEXPS in simulations. 

Benchmark. Recall that while the bound in (30) uses 
(1 — -) OPT as a benchmark, a more natural bench- 
mark would be the greedy optimum. We provide a 
preliminary convergence result for RankContextZoom, 
without any specific regret bounds. Such result is 
more elegantly formulated in terms of "anytime-Rank- 
ContextZoom" which uses the "anytime" version of 
ContextZoom (see Discussion 5.1). 

Theorem 5.5. Fix an instance of the k-slot MAB 
problem. The performance of anytime-RaxikContext- 
Zoom up to any given time t is equal to the greedy op- 
timum minus f[t) such that f{t) — > 0. 

Proof Sketch. It suffices to prove that with high prob- 
ability, anytime-RankContextZoom outputs a greedy 
ranking in all but fk{t) rounds among the first t 
rounds, where fk{t) — > 0. 

We prove this claim by induction on k, the number of 
slots. Suppose it holds for some fc — 1 slots, and focus 
on the fc-th slot. Consider all rounds in which a greedy 
ranking is chosen for the upper slots but not for the 
fc-th slot. In each such round, the fc-th slot replica of 
anytime-ContextZoom incurs contextual regret at least 
^/c, for some instance-specific constant 5k > 0. Thus, 
with high probability there can be at most Rk{t)/5k 
such rounds, where Rk(t) = o(i) is an upper bound 
on contextual regret for slot fc. Thus, one can take 
fk{t) = fu-i{t)+Ru{t)/5u. U 



Remark 5.6. Theorem 5.5 is about the "metric-less" 
setting from (Radlinski et al., 2008). It easily extends 
to the "ranked" version of any bandit algorithm whose 
contextual regret is sublinear with high probability. 
Discussion 5.7. It is an open question whether (and 
under which assumptions) Theorem 5.5 can be ex- 
tended to the "ranked" versions of non-contextual ban- 
dit algorithms such as RankUCBl. One assumption that 
appears essential is the uniqueness of the greedy rank- 
ing. To see that multiple greedy rankings may cause 
problems for ranked non-contextual algorithms, con- 
sider a simple example: 

• There are two slots and three documents xi , X2, xa 
such that /z = {\t\t\) and the relevance of each 
arm is independent of that of the other arms.^'^ 

An optimal ranking for this example is a greedy rank- 
ing that puts xi and X2 in the two slots, achieving 
aggregate click probability |. According to our intu- 
ition, a "reasonable" ranked non-contextual algorithm 
will behave as follows. The slot 1 algorithm will al- 
ternate between Xi and X2: each with frequency — )■ i. 
Since the slot- 2 algorithm is oblivious to the slot 1 
selection, it will observe averages that converge over 



time to 



■1 1 1\ 14 

■4' 4' 3'' 



SO it will select document x^ with 
frequency — )> 1. Therefore frequency — >■ 1 the ranked 
algorithm will alternate between {x,z) or (jj,z), each 
of which has aggregate click probability | . 

Desiderata. Wc believe that the above guarantees do 
not reflect the full power of our algorithms, and more 
generally the full power of conditional L-continuity. 
The "ideal" performance guarantee for RankBandit in 
our setting would use the greedy optimum as a bench- 
mark, and would have a bound on regret that is free 
from the inefficiencies outlined in Discussion 5.4. Fur- 
thermore, this guarantee would only rely on some gen- 
eral property of Bandit such as a bound on (contex- 
tual) regret. We conjecture that such guarantee is pos- 
sible for RankContextZoom, and (under some assump- 
tions) for RankCorrZoom and RankZoom. 

Further, one would like to study the relative benefits 
of the new "contextual" algorithms (RankContextZoom 
and RankCorrZoom) and the prior work (e.g., Rcink- 
Zoom). Discussion 5.7 suggests that the difference can 
be particularly pronounced when the pointwise mean 
has multiple peaks of similar value. We confirm this 



^^It is also wasteful (but perhaps less so) that we use a 
slot-fc bound for each slot i < k. 



^■^To make it more challenging for an algorithm, each 
document Xj can stand for a disjoint subset Xj of docu- 
ments with highly correlated payoffs. Further, some docu- 
ments in Xj can be far apart in the metric space. 

'^^Suppose Xj, j £ {1, 2} is chosen in slot 1. Then, letting 
S = {xj}, fi{xi\Zs) equals ii j = 1 and i otherwise 
(which averages to i), whereas fi{x3\Zs) = |. 
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experimentally in Section 6.3. 

6. Evaluation 

Let us evaluate the performance of the algorithms pre- 
sented in Section 4: "metric-oblivious" RankUCBl and 
RankEXPS, "metric- aware" non-contextual RankGrid- 
UCBl, RaiikGridEXPS and RankZoom, and contextual 
REinkContextZoom and RankCorrZoom. 

6.1. Experimental setup 

Using the generative model from Section 3 (Algo- 
rithm 1 with (9)), we created a document collection 
with \X\ = 2^5 « 32,000 documentsi'^ in a binary e- 
exponential tree metric space with e = 0.837. The 
value for e was chosen so that the most dissimilar doc- 
uments in the collection still have a non-trivial similar- 
ity, as may be expected for web documents. Each doc- 
ument's expected relevance ^i{x) was set by first identi- 
fying a small number of "peaks" j/i G X, choosing /x(-) 
for these documents, and then defining the relevance of 
other documents as the minimum allowed while obey- 
ing L-continuity and a background relevance rate ^o^ 



^j.{x) = max(/^o, h - mhi, 2?(.t, y^)). 



(31) 



For internal nodes in the tree, fj, is defined bottom-up 
(from leaves to the root) as the mean value of all chil- 
dren nodes. As a result, we obtain a set of documents 
X where each document x G X has an expected click 
probability /i(x) that obeys L-continuity. 

Our simulation was run over a 5-slot ranked bandit set- 
ting, learning the best 5 documents. We evaluated over 
300,000 user visits sampled from V per Algorithm 1. 
Performance within 50,000 impressions, typical for the 
number of times relatively frequent queries are seen by 
commercial search engines in a month, is essential for 
any practical applicability of this approach. However, 
we also measure performance for a longer time period 
to obtain a deeper understanding of the convergence 
properties of the algorithms. 

We consider two models for fj.{-) in (31). In the first 
model, two "peaks" {2/1,2/2} are selected at random 
with /x(-) = ^, and ^0 set to 0.05. The second model 
is less "rigid" (and thus more realistic): the relevant 
documents j/i and their expected relevance rates /i(-) 
are selected according to a Chinese Restaurant Process 
(Aldous, 1985) with parameters ?i = 20 and 9^2, and 
setting /.io = 0.01. The Chinese Restaurant Process is 
inspired by customers coming in to a restaurant with 
an infinite number of tables, each with infinite capac- 



^^This is a realistic number of documents that may be 
considered in detail for a typical web search query after 
pruning very unlikely documents. 



ity. At time t, a customer arrives and can choose to sit 
at a new table with probability 9/{t—l+6), and other- 
wise sits at an already occupied table with probability 
proportional to the number of customers already sit- 
ting at that table. By considering each table as equiv- 
alent to a peak in the distrubtion, this leads to a set of 
peaks with expected relevance rates distributed accor- 
ing to a power law. Following (Radlinski et al., 2008), 
we assign users to one of the peaks, then select rele- 
vant documents so as to obey the expected relevance 
rate /z(x) for each document x. 

As baselines we use an algorithm ranking the docu- 
ments at random, and the (offline) greedy algorithm 
discussed in Section 5.1. 

6.2. Experimental results 

Our experimental results are summarized in Figure 1 
and Figure 2. 

First, RankEXPS and RankUCBl perform as poorly as 
picking documents randomly: the three curves are 
indistinguishable. This is due to the large number 
of available documents and slow convergence rates 
of these algorithms. Other algorithms that explore 
all strategies (such as REC (Radhnski et al., 2008)) 
would perform just as poorly. This result is consis- 
tent with results reported by (Radlinski et al., 2008) 
on just 50 documents. On the other hand, algorithms 
that progressively refine the space of strategies ex- 
plored perform much better. 

Second, Making the UCBl-style algorithms "opti- 
mistic" (marked with a "-I-" after the algorithm name, 
see Section 4.1 for a discussion) improved performance 
dramatically. In particular, GridUCBl+ performed 
best of the non-contextual algorithms. RankZoom per- 
forms comparably to optimistic RankUCBl, and be- 
comes extremely effective if made optimistic. For the 
two contextual algorithms, we saw a similar increase 
in performance, hence we only show the performance 
of the optimistic versions. 

Third, RankCorrZooin+ achieves the best empirical 
performance, converging rapidly to near-optimal rank- 
ings. RankZoom+ is a close second. Interestingly, the 
theoretically preferred RankContextZoomdoes not per- 
form as well in simulations. This appears to be due 
to the much larger branching factor in the strategies 
activated by RankContextZoom slowing down the con- 
vergence. 

6.3. Secondary experiment 

As discussed in Section 5 (Discussion 5.7), some Rank- 
Bandit-style algorithms may converge to a suboptiinal 
ranking if fi has multiple peaks with similar values. To 
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investigate this, we designed a small-scale experiment 
presented in Figure 3. We generated a small collec- 
tion of 128 documents using the same setup with two 
"peaks" , and assumed 2 slots. Each peak corresponds 
to a half of the user population, with peak value /i = ^ 
and background value /io = 0.05. 

We see that RankContextZoom+ converges more slowly 
than the other zooming variants, but eventually out- 
performs them. This confirms our intuition, and sug- 
gests that RankContextZoom-f may eventually outper- 
form the other algorithms on a larger collection, such 
as that used for Figures 1 and 2. 

7. Proof of Theorem 3.1 

This section completes the proof of Theorem 3.1. First 
we state and prove a lemma needed to make the gener- 
ative model well-defined. Then we complete the proof 
sketch in Section 3.1 by proving (15). 

7.1. Extending fi from leaves to tree nodes 

Implicit in the construction in Section 3 is the following 
lemma which extends /i from documents (leaves of the 
document tree) to the internal nodes of the tree. 

Lemma 7.1. LetV be the shortest-paths metric of an 
edge-weighted rooted tree with node set V and leaf set 
X . Let /U : X — > [a, 6] he an L-continuous function on 
{X,T?). Then fx can be extended to V so that /x : V — >■ 
[a, 6] is L-continuous w.r.t. (V,!?). 



Proof. For each x G V, let C{x) be the set of all leaves 
in the subtree rooted at x. For each z <E £(y) the 
assignment /i(a;) should satisfy 

H{z) - V{x, z) < n{x) < n{z) + V{x, z) 
Thus /i(.T) should lie in the interval I{x) = 



[^ (x), /x+(x)], where 

fr{x) = SUp,g£(^) fi{z) - V{x, z), 

fj,+ {x) = ini^f^cix) m(2) +'D{x,z). 

This interval is always well-defined, i.e. tJ-~{x) < 
H^{x). Indeed, if not then for some z, z' € C{x) 

H{z) - V{x, z) > pi{z') + V{x, z') 
H{z) - fi{z') > V{x, z) + V{x, z') > V{z, z'), 

contradiction, claim proved. Note that /i+ (x) > a and 
fJ-~ (x) 5: b, so the intervals I{x) and [a, b] overlap. 

Using induction on the tree, we will construct values 
li{x), X £ V such that the Lipschitz condition 

\lJ'{x)- fJ.{y)\<D{x,y) ioT a.nx,y eX 

holds whenever a; is a parent of y. For the root xq, let 
fJ-{xo) be an arbitrary value in the interval I{xo)n[a, b]. 
For the induction step, suppose for some x we have 
chosen fi{x) S I{x)n[a, b] and y is a child of x. We need 
to choose fi{y) £ I{y) n [a, 6] so that \iJ.{x) — fi{y)\ < 
X'(.T,y). Note that 

^i{x) > fi^ {x) > sup3g£(,y) [ n{z) - V{x, y) - V{y, z) ] 

= l-i-^iy) -2?(x,y), 
fi{x) <l-L'^{x) <mi^^c{y) [Kz) + 'D{x,y)+'D{y,z)] 

^fi+{y)+Vix,y). 

It follows that /(y) and [iJ,{x) — 'D{x,y), ^{x)-\-'D{x,y)] 
have a non-empty intersection. Therefore, both inter- 
vals have a non-empty intersection with [a,b]. So we 
can choose fi{y) as required. This completes the con- 
struction of fi{) on V. 

To check that fi is Lipschitz-continuous on V , fix x, y G 
V, let P be the x ^f y path in the tree, and note that 






n 
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7.2. Conditional L-correlation: proof of (15) 

Let us re-introduce the notation from the proof sketch, 
in shghtly more detail. Fix documents x^y ^ X. We 
focus on the key event, denoted £, that no mutation 
happened on the x ^ y path. Recall that in Algorithm 
1, for each tree node u with parent v we assign ■k{u) ^r- 
Mu{7t{v)), where M„ : {0, 1} -^ {0, 1} is a random 
mutation which flips the input bit b with probability 
qb{u). If Mu is the identity function, then we say that 
no mutation happened at u. We say that no mutation 
happened on the a; — )■ y path if no mutation happened 
at each node in N^y, the set of all nodes on the x ^ y 
path except z. This event is denoted £; note that it 
implies 7r(x) = 7r(y) = 7r(z). Its complement E is, 
intuitively, a low-probability "failure event" . 

Fix a subset of documents S" C X. Recall that Zs 
denotes the event that all documents in S are irrele- 
vant, i.e. 7r(-) = on 5. Then (15) follows from the 
following lemma: 

Lemma 7.2. Vr[E\Zs\ < Pt[£] x (2/Pr[f]). 
Indeed, letting p ~ Pr[£] it holds that 

Pr[£|Zs]<min(l, ^) <3p. 

Remark. Lemma 7.2 inherits assumptions (7-8) on the 
mutation probabilities. For this Lemma, the upper 
bound (7) on mutation probabilities can be replaced 
with a much weaker one: 

ma,x{qo{u), qi{u)) < ^ for each tree node u. (32) 



In the rest of this section we prove Lemma 7.2. The 
proof is detailed, and quite heavy on notation. Curi- 
ously, we found this proof more intuitive if one actually 
thinks in terms of this (or similar) notation. 

Let Tu be the subtree rooted at a tree node u. For 
convenience, we will write u = b, b £ {0, 1} to mean 
7r(w) = b. 

In a sequence on claims, we will establish that 

Pt[Zs\z = 0]>Pt[Zs\z = 1]. (33) 

Intuitively, (33) means that the low-probability muta- 
tions are more likely to zero out a given subset of the 
leaves if the value at some fixed internal node is zero 
(rather than one). 

Before we prove (33), we use it to derive Lemma 7.2. 

7.2.1. Using (33) to prove Lemma 7.2 

Let us extend the notion of mutation from a single 
node to the x — > j/ path. Recall that N^y denotes 



the set of all nodes on this path except z. Then the 
individual node mutations {Af„ : u € N^y} collectively 
provide a mutation on N^y , which we define simply as 
a function M : N^y x {0, 1} -^ {0, 1} such that 7r(-) = 
M{-,tt{z)). Crucially, M is chosen independently of 
tt{z) (and of all other mutations). Let Ai be the set 
of all possible mutations of N^y By a slight abuse of 
notation, we treat the event £ as the identity mutation. 

Claim 7.3. Fix M e M and b e {0, 1}. Then 
Pt[Zs I M,TTiz) ^b]< PiiZs\£,TTiz) = 0]. 

Proof. For each tree node u, let Su = S Ci Tu be the 
subset of S that lies in the subtree Tu- Then by (33) 

Pr[Zs I A/, niz) ^ b] = JJu Pr[^s„ 1 7r(u) = A/(w, b)] 
< Uu Pr[^5„ I 7r{u) = 0] 

= PliZs\£,7T{z)^0], 

where the product is over all tree nodes u £ Nxy such 
that the intersection Su is non-empty. D 

Proof of Lemma 7.2. On one hand, by Claim 7.3 

Pr[Zs n£]^ Eb.M PrM Pr[^ = b] Pr[Zs | M, z = b] 
< Eb,M PrM Pr[2 = b] PiiZs \£,z = 0] 
^Pt[£] xPr[Zs| £, z^O], 

where the sums are over bits b £ {0, 1} and all muta- 
tions M £ M\ {£}. On the other hand, 

P^Zs] = Eb,M Pr[^/] Pr[^ = b] Pt[Zs \ M, z = b] 
(where the sum is over b £ {0,1} and M £ Ai) 

> Pt[£] Pt[z = 0] Pt[Zs \£, z = 0]. 
Since Pr[z = 0] > ^, it follows that 

Pt[£ I Zs] = Pt[Zs n £] I Px\Zs\ 



< 2 Pi-[£]/Pt[£] 



D 



7.2.2. Proof of (33) 

First we prove (33) for the case S C T, then we build 
on it to prove the (similar, but considerably more tech- 
nical) case S OTz =0. The general case follows since 
the events Zsn% a-nd Zs\t- arc conditionally indepen- 
dent given 7r(2:). 

Claim 7.4. If S cT then (33) holds. 

Proof. Let us use induction the depth of z. For the 
base case, the case x = y = z. Then S = {z} is the 
only possibility, and the claim is trivial. 
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For the induction step, consider children Ui of z such 
that the intersection 5"; = 30 7^. is non-empty. Let 
Ml , ... , u/j be all such children. For brevity, denote 
Zi = Zsi , and 

j/i(a|6) =Pr[u, = a|z = 6], a, 6 €{0,1}. 

Note that Vi{l,0) = qo{xi) and Wi(0,l) = qi{xi). 
Then for each 6 e {0, 1} we have 

Pr[Zs\z^b]^YltiP4Zr\z = b] (34) 

Pr[Zi \z^b]= Eae{o,i} ^»(«l^) P'^lZi I "« = a]. 

(35) 

By (34), to prove the claim it suffices to show that 

Pt[Z, I z = 0] > Pr[Z, \z = l] (36) 

holds for each i. By the induction hypothesis we have 

Pr[Z,\u^^O]>Pr[Z^\u,^l]. (37) 

Combining (37) and (32), and noting that by (35) we 
have i/j(0|0) > i^i(0|l), it follows that 

Pr[Z, I z = 0] - Pr[Z, | z = 1] 

= Eae{o,i} Pr[Z,\u,^a] ( j/i(a|0) - !/,(a|l) ) 
> Pr[Z, I u, = 1] Eae{o,i} ( Ma\0) " ^.(a|l) ) 
= 



where 



because l^^{0\Q) + Mi(l|0) = iyi{Q\l) + i^t{l\l) = 1. 



D 



Corollary 7.5. Consider tree nodes r,v,w such that 
r is an ancestor of v which in turn is an ancestor of 
w. Then for any c G {0, 1} 

Pr[?i = I w = 0, 7' = c] > Pr[u — 0\w — 1^ r — c]. 

Proof. We claim that for each b S {0, 1} 

PT[w = b\u = b]>PT[w^b\u=l-b]. (38) 

Indeed, truncating the subtree 7^, to a single node w 
and specializing Lemma 7.4 to a singleton set S = {w} 
(with z = u) we obtain (38) for & = 0. The case b = 1 
is symmetric. 

Now, for brevity we will omit conditioning on {r = c} 
in the remainder of the proof. (Formally, we will work 
on in the probability space obtained by conditioning 
on this event.) Then for each b £ {0, 1} 



Pr[u = I w = 6] 



Pr[u = OAw = b] 



Pt[u = a w = 5] U Pr[u = 1 A w = 6] 
1 



$(&) 



^ Pt[u = lAw = b] 



Pi[u-- 
Pr[u; 



OAw 

--b\u = 



-b] 
1] Pi[u = 1] 



Pr[w ^b\u^O] Pt[u ^ 0] 
is decreasing in b by (38). 



D 



We will also need a stronger, conditional, version of 
Lemma 7.4 whose proof is essentially identical (and 
omitted). 

Claim 7.6. Suppose S <Z Tz and u ^ z is a tree node 
such that Tu is disjoint with S . Then 

Pt:[Zs I z == 0, u = 1] > Pr[Zs | z = 1, u = 1]. (39) 

We will use Corollary 7.5 and Lemma 7.6 to prove (33) 
for the case 5 n T^ = 0. 

Claim 7.7. // S is disjoint with Tz then (33) holds. 

Proof. Suppose S is disjoint with 7^, and let r be the 
root of the tree. We will use induction on the tree to 
prove the following: for each c S {0, 1}, 

Pr[Z5 I r = c, z = 0] > Pr[Zs | r = c, z = 1] (40) 

For the induction base, consider a tree of depth 2, 
consisting of the root r and the leaves. Then z ^ S* is 
a leaf, so Zs is independent of 7r(z) given 7r(r), so (40) 
holds with equality. 

For the induction step, fix c S {0, 1}. Let us set up the 
notation similarly to the proof of Claim 7.4. Consider 
children Ui of r such that the intersection Si ^ SO Tm 
is non-empty. Let ui , ... , Ufc be all such children. As- 
sume z S Tui for some i (else, Zs is independent from 
7r(z) given 7r(r), so (40) holds with equality); without 
loss of generality, assume this happens for i = 1. For 
brevity, for a, fo e {0, 1} denote 

fi{a, b) = Pt[Zs, I Ml = a, z = 6] 
iyi{a\b) = Pr[ui = a | r = c, z ^ b]. 

Note that fi{a,b) and I'iialb) do not depend on b for 
i > I. 

Then for each b e {0, 1} 

Pr[Z5 \ r = c, z = b] 

= X! Jl ./i(ai,^) '^iloil^) 

= *xEae{oa} fi{a,b) i^i{a\b), 



where 



l + ^{b)' 



'^- E llMa^,b)ly,{a,\b) 

aiG{0,l},i>2 i>2 



Learning optimally diverse rankings over large document collections 



does not depend on of b. Therefore: 

Pi[Zs I r = c, z = 1] - Pt[Zs \r = c,z = l] 

= *xEae{o,i} 

[/i(a,0)i^i(a|0)-/i(a,l)^i(a|l)] (41) 

> * X Eae{o,i} /i(«' 1) [ M^\^) ~ Ma\l) ] (42) 

>$x/i(l,l)Ea6{o.i} [Ma\0)~Ma\l)] (43) 

= 0. (44) 

The above transitions hold for the following reasons: 

(41 ^42) By Induction Hypothesis, /i(a,0) > /i(a, 1) 

(42^43)By Lemma 7.6/1(0,1) > /i(l, 1), and more- 
over we have z/i(0|0) > i^i(0|l) by Corollary 7.5. 

(43^44) Since i^,(0|0)+i/,(l|0) = i.,(0|l)+i/,(l|l) = 1 
This completes the proof of the inductive step. D 

8. Further directions 

This paper initiates the study of online learning to 
rank in metric spaces, focusing on the "clean" simi- 
larity model (conditional L-continuity) . As discussed 
in Section 5, we conjecture that provable performance 
guarantees can be improved significantly. On the ex- 
perimental side, future work will include evaluating the 
model on web search data, and designing sufficiently 
memory- and time-efficient implementations to allow 
experiments on real users. An interesting challenge in 
such an endeavor would be to come up with effective 
similarity measures. An natural next step would be to 
also exploit the similarity between search queries. 

References 

Agrawal, R. (1995). The continuum-armed ban- 
dit problem. SIAM J. Control and Optimization, 
33(6):1926-1951. 

Aldous, D. J. (1985). Exchangeability and related top- 
ics. In Ecole d'Ete de Probabilites de Saint-Flour 
XIII, pages 1-198. 

Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002a). 
Finite-time analysis of the multiarmed bandit prob- 
lem. Machine Learning, 47(2-3):235-256. 

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, 
R. (20G2b). The nonstochastic multiarmed bandit 
problem. SIAM J. Comput, 32(l):48-77. 

Auer, P., Ortner, R., and Szepesvari, C. (2007). Im- 
proved Rates for the Stochastic Continuum-Armed 
Bandit Problem. In Conf. on Learning Theory 
(COLT), pages 454-478. 



Bartal, Y. (1996). Probabilistic approximations of 
metric spaces and its algorithmic applications. In 
IEEE Symp. on Foundations of Computer Science 
(FOCS). 

Bubcck, S., Munos, R., Stoltz, C, and Szepesvari, C. 
(2008). Online Optimization in X- Armed Bandits. 
In Advances in Neural Information Processing Sys- 
tems (NIPS), pages 201-208. 

Burges, C, Shaked, T., Renshaw, E., Lazier, A., 
Deeds, M., Hamilton, N., and Hullender, G. (2005). 
Learning to rank using gradient descent. In Intl. 
Conf. on Machine Learning (ICML), pages 89-96. 

CarboncU, J. and Goldstein, J. (1998). The use of 
MMR, diversity-based reranking for reordering doc- 
uments and producing summaries. In ACM Intl. 
Conf. on Research and Development in Information 
Retrieval (SIGIR), pages 335-336. 

Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, 
learning, and games. Cambridge Univ. Press. 

Chen, H. and Karger, D. R. (2006). Less is more: 
Probabilistic models for retrieving fewer relevant 
documents. In ACM Intl. Conf. on Research 
and Development in Information Retrieval (SICIR), 
pages 429-436. 

Chu, W. and Ghahramani, Z. (2005). Gaussian pro- 
cesses for ordinal regression. J. of Machine Learning 
Research, 6:1019-1041. 

Fakcharoenphol, J., Rao, S., and Talwar, K. (2004). A 
tight bound on approximating arbitrary metrics by 
tree metrics. J. of Computer and System Sciences, 
69(3):485-497. 

Gupta, A., Krauthgamer, R., and Lee, J. R. (2003). 
Bounded geometries, fractals, and low-distortion 
embeddings. In IEEE Symp. on Foundations of 
Computer Science (FOCS). 

Hazan, E. and Megiddo, N. (2007). Online Learning 
with Prior Knowledge. In Conf. on Learning Theory 
(COLT), pages 499-513. 

Kleinberg, R. (2004). Nearly tight bounds for the 
continuum-armed bandit problem. In Advances 
in Neural Information Processing Systems (NIPS), 
pages 697-704. 

Kleinberg, R., Niculescu-Mizil, A., and Sharma, Y. 
(2008a). Regret bounds for sleeping experts and 
bandits. In Conf. on Learning Theory (COLT), 
pages 425-436. 



Learning optimally diverse rankings over large document collections 



Kleinberg, R. and Slivkins, A. (2010). Sharp 
Dichotomies for Regret Minimization in Metric 
Spaces. In ACM-SIAM Symp. on Discrete Algo- 
rithms (SODA), pages 827-846. 

Kleinberg, R., SUvkins, A., and Upfal, E. (2008b). 
Multi- Armed Bandits in Metric Spaces. In ACM 
Symp. on Theory of Computing (STOC), pages 681- 
690. 

Pandey, S., Agarwal, D., Chakrabarti, D., and Josi- 
fovski, V. (2007). Bandits for Taxonomies; A Model- 
based Approach. In SIAM Intl. Conf. on Data Min- 
ing (SDM). 

Radlinski, F., Kleinberg, R., and Joachims, T. (2008). 
Learning diverse rankings with multi-armed bandits. 
In Intl. Conf. on Machine Learning (ICML), pages 
784-791. 

Slivkins, A. (2009). Contextual Bandits with Similar- 
ity Information, http://arxiv.org/abs/0907.3986. 

Streeter, M. and Golovin, D. (2009). An online al- 
gorithm for maximizing submodular functions. In 
Advances in Neural Information Processing Systems 
(NIPS), pages 1577-1584. 

Taylor, M., Guiver, J., Robertson, S., and Minka, T. 
(2008). Softrank: optimizing non-smooth rank met- 
rics. In ACM Intl. Conf. on Web Search and Data 
Mining (WSDM), pages 77-86. 



